forATM - Based High Performance DistributedComputing Environments

Current advances in processor technology and the rapid development of high-speed networking technology (e.g., Asynchronous Transfer Mode (ATM), Myrinet, and Fast Ethernet) have made network-based computing an attractive environment for large-scale High Performance Distributed Computing (HPDC) applications. However, due to the communication overhead between computers and the in exible communication architectures of the parallel/distributed software tools, most HPDC applications do not fully utilize the bene ts of high-speed communication networks. This can be mainly attributed to the high cost associated with system calls and context switching, redundant data copying during protocol processing, lack of support to overlap computation and communication at the application level, and tight coupling of data and control functions. In this paper, we present an architecture, implementation, and performance evaluation of a multithreaded message-passing system for an ATM-based HPDC environment that we call the NYNET (ATM wide-area network testbed in New York state) Communication System (NCS). NCS uses multithreading to provide e cient techniques for overlapping computation and communication. By separating control and data activities, NCS eliminates unnecessary control transfers over the data path. This optimizes the data path and improves the performance. NCS supports several di erent ow control algorithms, error control algorithms, and multicasting algorithms. Furthermore, NCS allows programmers to select at runtime the suitable communication schemes per-connection basis to meet the Quality of Service (QOS) requirements of a given application. NCS provides three application communication interfaces: Socket Communication Interface (SCI), ATM Communication Interface (ACI), and High Performance Interface (HPI) to support various classes of applications. The SCI is provided mainly for applications that must be portable to many di erent computing platforms. The ACI provides services that are compatible with ATM connection oriented services where each connection can be con gured to meet the QOS requirements of that connection. This allows programmers to fully utilize the bene ts of the ATM network. The HPI supports applications that demand low-latency and high-throughput communication services. In this interface, NCS uses read/write trap routines to reduce latency, data transfer time, and avoid using traditional communication protocols. We analyze and compare the performance of NCS with those of other message-passing systems such as p4, PVM, and MPI in terms of point-to-point, multicasting, and application performance. The benchmarking results show that NCS outperforms other message-passing systems and provides exible communication services for various classes of applications.